System for determining and tracking movement during a medical procedure

ABSTRACT

An image guidance system for tracking a surgical instrument during a surgical procedure. The image guidance system includes a plurality of cameras adapted to be located external to a surgical area for capturing images of optically visible patterns. A processing system receives and processes the images to recognize the patterns and triangulate the locations and orientations of each pattern relative to a camera. The processing system uses a reference dataset which defines a reference coordinate system based on alignment to a portion of the oral anatomy. The processing system determines the location and orientation of the tracked instrument based on the reference dataset.

RELATED APPLICATION

This Application is related to co-pending application Ser. No.14/488,004, filed on Sep. 16, 2014.

FIELD OF THE INVENTION

The present invention relates to a system for image guided surgery and,more particularly, to an system for determining and tracking movementduring a medical procedure using an externally visible reference point.

BACKGROUND

Image guided surgery has had extensive developments over the years andis now a very important tool in surgical procedures. Most of thedevelopments have centered around imaging locations in the body wherethere is very little access, such as internal organs.

Oral surgery, which is defined herein as any surgery occurring withinthe oral cavity, can be just as difficult to conduct visually. The oralcavity is relatively small and difficult for a patient to maintain openfor prolonged periods of time. Even if a surgical site is visible, oncethe drill penetrates, it becomes difficult to determine where the tip isat any given time.

Image guided surgery involves the use of a computed or computerizedaxial tomography scan, commonly referred to as CT or CAT scans, tocreate a digital image of the surgical site (typically in threedimensions). The surgeon then creates a plan for the surgery using theimage. During surgery, the image generated from the prior CT scan isused in conjunction with a special instrument, to visually depict wherethe tip of the instrument is inside the patient.

In order to do so, the digital image from the scan must be accuratelyregistered to the surgical site of the patient such that movement of thepatient causes adjustment of the digital image. The exact location ofthe instrument tip relative to the patient must also be known.

For oral surgery, such as during dental implant placement, a doctor hasto drill in free space while controlling the drill in six degrees offreedom with the patient potentially moving. This makes accuratelydrilling into good bone while avoiding roots and nerves very difficult.As such, image guided surgery has recently been used to facilitate thedrilling process. CT scans of the patient's teeth are used by thedoctors to accurately determine bone density, width and height, as wellas understand relationships of other teeth and anatomical structures inorder to plan a surgical event to provide the restorative solution thatwould likely be the most successful and least traumatic.

Planning software and fabrication systems exists today that uses the CTimage to assist in translating a pre-surgical plan to a passive surgicalguide, i.e., creating a virtual plan for the surgery and thenprefabricating in the dental laboratory a surgical guide to implementthe plan. These passive surgical guides help accurately direct thedoctor to the proper location, angle and depth. Passive image guidedsurgery has limitations. They must be fabricated prior to surgery in adental lab or by a guide manufacturer. This requires greater doctor andpatient time and expense. If there is a change in a patients mouth orthe doctor desires to change the plan, the guide is no longer useful. Inmany cases the patient is unable to open their mouth wide enough toaccommodate the instruments needed and the guide.

Active image guided surgery solves many of the problems of passivelyguided systems, i.e., limited maximal mouth opening, the need toprefabricate a passive guide and the inability to change the plan duringsurgery can be overcome by actively guided systems. In order to provideactive image guided surgery, the position of the patient's mouth,specifically the bone and teeth, must be accurately tracked andregistered to the scanned image and the surgical tool. In order to doso, most conventional systems require the creation of a registrationdevice that is attached to the patient's head or inserted into the mouthwhich includes fiducial markers and a sensor. Some registration devicesare attached to the outside of the head, for example, a head mountedfixture. Others involve a fixture that is attached to the jawbone withthe sensors located outside the mouth in order to limit the interferencewith the surgical zone and to permit optical sensors to track themovement of the fixture and surgical tool.

In order to create the oral fixture, an impression is taken, typicallyof both the upper and lower sets of teeth weeks in advance of theoperation. The impression is then sent to a lab where a cast is madesubstantially duplicating the teeth. From the cast an oral fixture ismade that either seats on the teeth or is designed to be drilled intothe jawbone. The fixture includes at least the fiducial markers andalso, if not fitted with a sensor, includes mounting locations for theoptical sensors.

After the lab creates the fixture it is sent back to the dental surgeon.The patient is brought in, fitted with the fixture and a CT scan istaken. The patient is once again sent home. A digital image of thepatient's oral cavity is created from the scan and the surgeon developsthe surgical plan.

The patient is then brought in for the operation. The fixture isattached to the patient. Optical transmitters are located about thepatient and emit signals that are detected by the sensor(s). Thesensor(s) send a signal to the software as the patient's mouth moves andan adjustment is made to the digital image of the patient's oral cavity.The software also tracks the position of the instrument and depicts animage of the instrument in the proper location relative to the digitalimage of the teeth.

In addition to the inconvenience to the patient, existing systems tendto have some difficult accurately registering the patient to the digitalscan. All present dental active image-guided surgery systems involve theuse of optical tracking which requires that the fixture that is placedin the patient's mouth extends outside the mouth in order to be detectedby the optical transmitter or receivers.

SUMMARY OF THE INVENTION

An image guidance system is disclosed for tracking a surgical instrumentduring oral surgery. The system includes a fixture configured to beremovably attached to a patient's anatomy in a location near a surgicalarea.

A first tracking assembly, including a bracket assembly, is removablyattached to the fixture. The first tracking assembly includes a firsttracking pattern surface including a first optically visible pattern.The bracket assembly positions the first tracking pattern at a locationspaced apart from the surgical area.

The system includes a tool for use in the surgical procedure. A secondtracking assembly is attached to the tool and includes a second trackingpattern surface. The second tracking pattern surface includes a secondoptically visible pattern.

A plurality of cameras are mounted away from the surgical area at aposition that permits the cameras, when activated, to capture images ofthe optically visible patterns on the first and second tracking patternsurfaces.

A processing system is connected to the cameras and processes thecaptured images. The processor is configured to recognize the opticallyvisible patterns and triangulate the locations and orientations of thefirst and second tracking assemblies. The processing system determinesthe location and orientation of the tracked tool based on a referencedataset that includes the location and orientation of the fixture withrespect to a CT scan. The processing system analyzing relativetransforms between each camera of the optically visible patterns on thefirst and second tracking pattern surfaces.

The fixture is preferably configured to removably attach to one or moreteeth in a patient's mouth.

The bracket assembly includes a bracket mount that removably attaches toflanges on the fixture. The bracket assembly also includes a trackingmount that attaches to the first tracking pattern surface, and a supportarm that attaches the bracket mount to the tracking mount. The bracketmount preferably includes two spaced apart mounting posts that engagewith the flanges. vvThe attachment of the tracking mount to the firsttracking pattern surface is preferably adjustable.

In one embodiment, the tracking mount includes a base with a series ofindentations and protrusions. The first tracking assembly includes aframe which attaches to the tracking mount so as to be adjustablyoriented with respect to the bracket assembly. The frame may include aseries of indentations and protrusions that are configured to mate withthe indentations and protrusions on the tracking mount so as to permitrotation of the tracking frame relative to the base.

The optically visible patterns each preferably contain a plurality of 2Dcontrasting shapes, the contrasting shapes arranged so as to uniquelydifferentiate each optically visible pattern from the other opticallyvisible pattern.

Each camera is located so as to capture an image of each opticallyvisible pattern which is at a different viewing angle than the image ofthe same optically visible pattern captured by the other camera(s). Thecameras send data representing the 2D images of the optically visiblepatterns.

The reference data includes location data for contrasting shapes on aplurality of patterns. The processor uses the reference data todetermine the specific pattern in the image.

The foregoing and other features of the invention and advantages of thepresent invention will become more apparent in light of the followingdetailed description of the preferred embodiments, as illustrated in theaccompanying figures. As will be realized, the invention is capable ofmodifications in various respects, all without departing from theinvention. Accordingly, the drawings and the description are to beregarded as illustrative in nature, and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

For the purpose of illustrating the invention, the drawings show a formof the invention which is presently preferred. However, it should beunderstood that this invention is not limited to the precisearrangements and instrumentalities shown in the drawings.

FIG. 1 is a perspective view of a tracking assembly with a patterndisplay surface attached to an oral fixture for use in an embodiment ofthe present invention.

FIG. 2 is a top view of the oral fixture and tracking assembly of FIG.1.

FIG. 3 is a perspective view of another embodiment of a trackingassembly with a pattern display surface attached to an oral fixtureaccording to the present invention.

FIG. 4 is a side view of the oral fixture and tracking assembly of FIG.3.

FIG. 5 is a side view of a bracket for attaching a tracking assembly toan oral fixture according to the present invention.

FIG. 6 is a different side view of the bracket of FIG. 5.

FIG. 7 is a perspective view of another embodiment of a trackingassembly attached to a dental instrument according to the presentinvention.

FIG. 8 is a side view of the dental instrument and tracking assembly ofFIG. 7.

FIG. 9 is the tracking assembly of FIG. 7 removed from the dentalinstrument.

FIG. 10 is an illustration of one embodiment of a pattern for use on thepattern display surface of the tracking assembly.

FIG. 11 is an illustration of a tracking tile with four trackingpatterns arranged for unique tracking of the pattern.

FIG. 11A is an enlarged view of a portion of the tracking pattern ofFIG. 11.

FIG. 12 is a flow chart illustrating a conventional method ofdetermining model pose.

FIG. 13 is a flow chart illustrating a method of determining model poseaccording to the present invention.

FIG. 14 is a schematic representation of an oral fixture and dentalinstrument with imaging cameras according to the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention addresses the prior art deficiencies by providingan image guidance system for efficiently tracking a patient's movementduring surgery. The present invention will be described as it related tooral surgery and the tracking of the movement of a patient's mouth, butthe invention is not necessarily limited to that embodiment. In oneembodiment, the image guidance system includes a plurality of cameraslocated outside the oral cavity to provide images of optically visiblepatterns attached to the patient through an oral fixture and that arelocated external to the area being operated on. The images are used todetect and tracking movement of the patient's mouth, and/or a surgicalinstrument or tool. A processing system receives and processes theimages to recognize patterns and triangulate the locations andorientations relative to each camera. The processing system uses areference dataset which defines a reference coordinate system based onalignment to a portion of the oral anatomy. The processing systemdetermines the location and orientation of the tracked surgicalinstrument and the oral fixture based on the reference dataset.

Turning now to the figures, embodiments of the image guidance system 10are shown for use in an oral surgical procedure. As will becomeapparent, the inventive features are not limited to oral surgicalprocedures and have applicability to other surgical procedures. In oneembodiment the system 10 includes an oral dental appliance or fixture 12that is designed to attach to one or more teeth of the patient. Onesuitable fixture is described in co-pending application Ser. No.14/209,500, the disclosure of which is incorporated herein by referencein its entirety. Details of the fixture 12 as referenced herein can befound in that application. The fixture 12 is preferably removablyattachable to the patient's teeth and includes a support 14 that is madefrom a suitably strong material, preferably a thermoset plasticmaterial, that is sufficiently rigid so as not to deform when subjectedto the elevated temperatures discussed below. In one embodiment, theplastic material is polyphenylsulphone or acetal copolymer. The support14 includes a base 16 that is, preferably, generally planar, with aninner wall 18 and an outer wall 20. The inner wall 18 and outer wall 20are attached to and extend outward from the base 16. Preferably thewalls 18, 20 extend outward from the base 16 at substantially orgenerally right angles from the base 16. However as will be appreciatedthe walls could be at other desired angles from the base 16. The wallsand base are preferably formed as an integral component. The spacing ofthe inner and outer walls 18, 20 is larger than the width of the teethto which the oral fixture 12 is intended to be attached. It should bereadily apparent that the spacing of the walls 18, 20 can be differentbetween fixtures designed for adults and children. The walls 18, 20preferably have a height from the base which extends below the top ofthe patient's teeth when installed. Preferably the height is sufficientto extend about 10 mm to about 13.5 mm down from occlusal surface wheninstalled on a patient's tooth with the overlying material.

As described in co-pending application Ser. No. 14/209,500, the oralfixture 12 also includes a moldable thermoplastic material located on aninner surface of the support 14, preferably on the base 16. The moldablematerial is designed to form an impression of a portion of a patient'steeth. More specifically, when the moldable material is in its uncured(unset) state, the material is “activated” by placing the oral fixture12 (support 14 with moldable material on it) into a bowl of warm or hotwater that is at a temperature above which the material begins to becomemoldable. Preferably the chosen material has a characteristic thatprovides the user with a visual indication that the material is ready tobe molded, such as changing color (e.g., from white to clear ortranslucent). Once the material is activated, the oral fixture 12 isplaced on a patient's teeth and slight downward pressure is appliedcausing the moldable material to deform around the top and at least someof the sides of the teeth between the support walls 18, 20. After aprescribed period of time, generally about 30 seconds to one minute, themoldable material sets to form an impression of the outside shape andcontours of the teeth that were covered by the material. The oralfixture 12 can then be removed from the patient's mouth. Further curingcan be achieved by placing the oral fixture 12 with the mold materialinto a bowl of cold or ice water to complete the setting process.

The material selected must remain solid (cured) at temperaturestypically existing in a person's mouth (generally, around 100 degreesF.), and moldable at a temperature above that (e.g., above 130 degreesF.), at least until it is initially set. The material should besufficiently rigid in its cured state so as to maintain the shape of theimpression without distorting. Suitable thermoplastic materials for usein the invention includes Polycaprolactone or Polyvinylsiloxane (PVS).However, any type of moldable material that can set and retain animpression can be used in the present invention. The moldable materialmay be flavored to please the patient during the molding process. Theamount of material used will vary depending on the number and size ofteeth that are to be molded.

The oral fixture 12 also includes a plurality of fiducial markers 80mounted on the support 14 in order for the system to determine where theoral fixture 12 (and thus the camera) is relative to the patient'steeth. The markers 80 are at certain locations on the fixture 12 and arepart of a registration system for properly locating the fixture 12 inspace. As will be discussed in more detail below, the fiducial markersare detected during a CT scan of the patient's mouth and their locationis registered in the scan. There are preferably at least three fiducialmarkers 80 spaced apart from each other and rigidly attached to thesupport 14. The use of the three fiducial markers permits location oforal fixture in three dimensions. The fiducial markers may be located onthe base 16 and/or the walls 18, 20.

The fiducial markers 80 may be spherical in shape and/or colored so asto be easily detected by a technician or doctor, as well as the softwarebeing used. More specifically, in order for the fiducial markers 80 tobe detected in a scanned image, the fiducial markers 80 must have adifferent radiodensity (i.e., the density that is detected by the CTscan) than the fixture, moldable material and teeth. In one embodiment,the fiducial markers 80 are ceramic ball bearings. However, othermaterials, shapes and sizes may be used. Preferably the fiducial markers80 each have their own radiodensity or are of different sizes or shapesso that a software program can be used to automatically detect thedifferent fiducial markers 80 in the scanned image. The software mayalso apply a color in the scanned image that corresponds to the markerscolor or shape to assist in registration of the oral fixture 12 as willbe discussed further below. It is also contemplated that the fiducialscan include passive optical attributes, such as specular or diffusesurfaces, or active optical attributes, such as light emittingmaterials, for use in visually locating the fiducials relative to acamera or other location.

While the preferred fiducial markers are distinguished from the teethand oral fixture 12 by their radiodensity, it is also contemplated thatother distinguishing features can be used other than density. Forexample, the markers can be pre-fixed transmitters or other positionlocation devices.

The oral fixture 12 also includes at least one mount 26 attached to orformed integral with the support 14. In the illustrated embodiment, themount 26 extends outward from the outer wall 20. As will be discussedbelow, the mount 26 is configured to have a tracking assembly 200attached to it for use in tracking motion (position changes) of thefixture 12. In one embodiment, the mount 26 includes at least one flange28 and more preferably two spaced apart flanges 28, 30 that extend outof the side of the fixture 12. Each flange 28, 30 may include notches orindentations 32 formed in the opposite lateral sides of the flange 28,30.

A bracket assembly 100 is removably attachable to the mount 26 of theoral fixture 12 and is configured to hold the tracking assembly 200. Inthe illustrated embodiment, the bracket assembly includes a bracketmount 102 that removably attaches to the flanges 28, 30 on the fixture,a support arm 104, and a tracking mount 106. The bracket mount 102includes two spaced apart mounting posts 108 _(A), 108 _(B). Eachmounting post 108 preferably includes a protrusion 110 that isconfigured to engage with and sit in the notch 32 such that the mountingposts 108 _(A), 108 _(B) are positioned on either side of and againstthe flanges 28, 30.

The support arm 104 includes a main portion 112 and a fixture portion114 that extends between the posts 108 _(A), 108 _(B). In oneembodiment, the support arm 104 is rigidly, preferably fixedly, attachedto one of the posts 108 _(A). The other post 108 _(B) (the one furthestfrom the main portion 112) is preferably slidably disposed on thefixture portion 114 so that the spacing between the posts 108 _(A), 108_(B) is adjustable. A distal end of the fixture portion 114 extendsthrough the post 108 _(B). Threads (not shown) are preferably formed onthe distal end of the fixture portion 114. A knob 116 is threaded ontothe distal end of the fixture portion. As shown in FIG. 6, tracker armpins 113 are used to attach the posts 108 captive on the support arm112. This allows the posts 108 to rotate freely about the support arm112. When mounted to the fixture 12, post 108 _(A) is positioned againstthe flanges 28, 30 so that the protruding portion 110 seats in the notch32. The other post 108 _(B) is slid on the distal end of the fixtureportion of the support arm 104 until its protruding portion 110 seatswithin the other notch 32. The knob 116 is tightened, thereby securingthe arm 104 to the oral fixture 12. It should be readily apparent thatthe posts 108 could, instead, include notched portions and the flanges28, 30 could have protruding portions, or the posts and flanges mightsimply have flush mounting surfaces.

As discussed above, the opposite end of the arm 104 includes a trackingmount 106 for attaching a fixture tracking assembly 200. In theillustrated embodiment, the tracking mount 106 includes a threaded stub118 and a base 120. The base 120 preferably has a series of teeth orindentations and protrusions 122. The base 120 and threaded stub 118 arepreferably integral with the main portion 112 of the arm 104.

The fixture tracking assembly 200 is attached to the tracking mount 106so that it is preferably adjustable. More particular, the fixturetracking assembly 200 includes a frame 202 which attaches to thetracking mount 106 of the bracket assembly 100. The attachment ispreferably configured to permit the frame to adjustably oriented withrespect to the bracket assembly 100 as will be discussed in more detail.In the illustrated embodiment, the frame 202 includes a hole 203 (shownin FIG. 4) with threads that threadingly engage with the threads on thestub 118 of the arm 104. Preferably there are a series of teeth orindentations and protrusions 204 that are configured to mate with theteeth or indentations and protrusions 122 on the bracket assembly 100.The inclusion of the mating teeth 122/204 permits accurate andrepeatable adjustability of the position of the frame 202 relative tothe support arm 104. In the illustrated embodiment, the mounting of thefixture tracking assembly 200 to the bracket assembly 100 permits thetracking assembly to be lockably positioned at different positions ofrotation about axis 206.

The tracking assembly includes a pattern display surface 208 that isattached to or formed on the frame 202. By adjusting the attachment ofthe fixture tracking assembly 200 to the bracket assembly 100, it ispossible to change the orientation of the pattern display surface 208about the axis 206. This is a beneficial feature since it permits thepattern display surface 208 to be oriented at a suitable position duringuse so as to provide maximum detectability of the surface by externallymounted cameras.

The pattern display surface can have any suitable shape. In oneembodiment shown in FIGS. 1 and 2, the pattern display surface 208 ofthe tracking assembly is substantially cylindrical having an axis thatis preferably collinear with the axis 206. In another embodiment shownin FIGS. 3 and 4, the pattern display surface 208 of the trackingassembly is substantially flat or planar. It should be readily apparentthat any other shape could be used with the present invention.

A tracking pattern 210 is disposed or formed on the pattern displaysurface 208. The tracking pattern 210 is an optically visible patternthat is configured to provide visual reference points for externallymounted cameras to detect for use by a computer system to track theposition and movement of the tracking assembly, and, thus, the oralfixture 12. In an embodiment, the tracking pattern may include a seriesof non-repetitive Quick Reference or QR Codes spaced apart on thesurface of the tracking assembly 200. Application Ser. No. 14/209,500describes some suitable tracking patterns that can be used in thepresent invention. FIG. 10 illustrates a 2D tracking pattern that may beused in the present invention.

Bar codes, Aztec codes or other 2D codes, or graphical images, couldalso be used. The pattern preferably uses contrasting colors, such asblack (shown in dense cross-hatching) and white, to facilitate detectionand recognition by the system. The arrangement of the checkerboardsquares are arranged so as to be easily and quickly identified. It isalso contemplated that other mechanisms can be used to provide thereference data needed, including LEDs, a data matrix, data glyphs, orraised or lowered features similar to braille. The tracking pattern 208may be formed on a layer of material that is adhered to the frame of thetracking assembly. Alternatively, the tracking pattern may be molded oretched onto or disposed directly on the frame.

It is contemplated that the fixture tracking assembly 200 may beconfigured to provide backlighting or other mechanism to increase thecontrast of the tracking pattern 210 in order to facilitate detection.If the tracking assembly is backlit, the tracking pattern 210 ispreferably made of at least partially transparent or translucentmaterial so as to enhance the contrast. It is also contemplated that afluorescent material can be used to facilitate detection.

Referring now to FIGS. 7-9, a surgical tool tracking assembly 300according to one embodiment is shown mounted to or part of a surgicaldental tool 302, such as a drill. The tool tracking assembly 300includes a tool mount 304 that is designed to secure a tool patternsurface 306 to the tool 302. The tool mount 304 includes an opening 308that fits around the body 310 of the tool 302 in a secure manner so thatthe tracking assembly moves in combination with the surgical tool. Theattachment could be through any of a number of different mechanisms wellknown in the art. For example, the tool tracking assembly is attached,for example, with a collet or similar well known mechanism which may beremovably screwed on or clamped down on the tool body so as to securethe tool tracking assembly to the tool. A hole may be included to permitirrigation tubes and tool camera wires.

A tool tracking pattern 308, similar to the fixture tracking pattern210, is disposed or formed on the tool pattern surface 306. The tooltracking pattern 308 is an optically visible pattern that is configuredto provide visual reference points for externally mounted cameras todetect for use by a computer system to track the position and movementof the tool tracking assembly 300. The pattern shown in FIG. 10 could beused as the tool tracking pattern.

Referring now to the FIG. 11, an embodiment of a tracking tile 400 isshown. In this embodiment, the tracking tile 400 includes a portion ofthe tracking pattern 210. More specifically, in the illustratedembodiment, when four tracking tiles are arranged as shown in FIG. 11,the intersection of the four tiles defines the tracking pattern 210 asindicated by the dashed lines. In the illustrated embodiment the lightlycross-hatched boxes can be either black or white within the scope of theinvention. The choices of coloring of the lightly cross-hatched boxes,in combination with other boxes on the tile 400 permit the tile to beuniquely defined so that the pattern on the individual tile 400 isrecognized by the system.

There are several benefits to using the tracking tile 400. First, eachtile includes, on average, approximately 50% intensity (i.e., 50% lightand 50% dark). This facilitates the ability of a computer systemdetecting, through a camera, the boxes in the tile by permitting thecomputer system to adjust the gain and exposure of the cameras in orderto maximize detection performance. Also, when four tiles 400 arearranged as shown in FIG. 11, each tracking tile 400 includes a minimumof thirteen defined points 402, which in the preferred embodiment arex-corners, the center point of two intersecting lines between adjacentboxes of opposed color (i.e., black (dense cross-hatching) and white).The advantage of choosing x-corners as the defined points, is thatlocation of the center point can be located to sub-pixel accuracy, andthe location is stable under typical image degradations, in particularover-illumination and under-illumination and sensor noise, however it iscontemplated that other types of defined points and combinations ofdifferent types of defined points can be used, for instance thecentroids of circles or other shapes, and corner points on shapes withangled contrast regions. More particularly and with reference to FIG.11A, which is an enlarged view of four adjacent boxes of opposed color,adjacent boxes of opposed colors (404 white, 406 black (densecross-hatching)) are separated from one another by a line 408. In oneembodiment, the system is programmed to detect two distinct colors, inthis case, black and white, and locate a line between adjacent boxes ofthose two colors. For example, when the system detects two adjacentdistinct colors it seeks a series of two or more adjacent points A, Bbetween those distinct color boxes and defines a line 408 between theseries of points A, B and, thus, between the two adjacent blocks 404,406. The system analyses the pattern in order to detect four adjacentboxes of alternately distinct colors that form a square as shown in FIG.11A. The intersection of the lines 408 between the boxes cross at adefined point 402. An alternate method for detecting an x-corner in animage is through analysis of the image structure tensor as in the Harriscorner detector, well-known to those skilled in the art. In its broadestembodiment, each tile is any uniquely-identifiable (unambiguous) subsetof a pattern. Also it is contemplated that tiles can overlap with othertiles, and do not need to be shaped as a series of squares, but can beoddly-shaped.

In embodiments with defined points that are not x-corners, an alternatedetection algorithm, sensitive to the particular type of defined pointcan be used. For example, if the defined points include centroids ofcircular features, algorithms such as Laplacian of Gaussians, Differenceof Gaussians, or Determinant of Hessians can be used.

As discussed above, the when four tiles 400 are arranged as shown inFIG. 11, each tracking tile 400 includes a minimum of thirteen definedpoints 402. The system includes a lookup table or stored data on aplurality of patterns, including size and arrangement (e.g., location)of boxes and defined points in the various patterns. The system alsoincludes a transform (transformation matrix) for converting the patterndata to a target coordinate system. Preferably, the transform is a rigidbody transform or a 3D affine transformation which includes 3D rotation,translation, scaling, and skew. It is contemplated that the transformcan include non-linear deformations to conform the arrangement to anon-planar surface.

More specifically, in one embodiment, each tile has the followingcharacteristics: (i) it contains a square grid of two (or more) distinctcolors (preferably black and white), (ii) the defined points appear onlyat the grid locations (intersections), and (iii) are printed on a planarsurface, which means that under perspective imaging (i.e., when observedin an arbitrary orientation by a camera recording an image), the tileappears deformed by a locally-affine transformation (meaning that theprinted square tile will appear stretched and skewed into a rhombusshape in the image).

In the case where a planar tile is used (i.e., a tile where the gridsare printed on a planar surface), such as the pattern tile arrangementin FIG. 3, each defined point is analyzed by the system as follows:

-   -   a. Adjacent defined points are located. More specifically, in        one embodiment adjacent defined points can be a simple near        neighbor (based on Euclidean distance) to the defined point. In        another embodiment, the neighbor distance can be replaced by the        distance along a high-contrast edge. Many other distance        functions are contemplated    -   b. Using the defined point being analyzed and the adjacent        defined points, a pair of basis vectors between the defined        point and two of its adjacent defined points is determined,    -   c. The basis vectors are then used to compute a rectifying        affine transform that will transform a rhombohedral image patch        into a square image patch, three of whose corners are the        defined point and its two adjacent defined points (i.e., a        transform to convert the detected defined point locations in the        image into the square grid used in a planar printed tile.)    -   d. The system then uses the affine transform, assuming that the        three detected defined points are at the corner and edges of a        tile, to predict where each grid location will be in the image        (essentially creating an overlay on the image of the grid skewed        according to the affine transform).    -   e. The image is analyzed at each predicted grid location to        calculate a descriptor. A descriptor describes a local region.        In one implementation, it is a 9×9 matrix representing a tile,        where each element of the matrix is an x-corner type. The        x-corner type applied to a local 2D coordinate system, i.e., the        basis vectors define a local coordinate system which permit the        defining of “right” and “up”. In this coordinate system, if the        image patch in the upper-left is bright (and the upper right is        dark), the x-corner is left-oriented, if the opposite pattern        appears, it's right-oriented. As such, each element of the        matrix is either Left-Oriented, Right-Oriented, or no X-corner        detected).    -   f. A score is then computed as to how closely the descriptor        matches a stored encoding scheme. In the present invention,        x-corners may be detected on parts of the scene that are not        within the pattern. Likewise, many chosen combinations of three        adjacent defined points may not in fact correspond to the tile        corner and edges. In order to analyze for these false detections        is to verify that the structure of the x-corner is consistent        with the internal relationships defined by the encoding scheme        that is chosen. In the preferred implementation, there are        defined relationships between x-corners at various grid        locations (e.g., each tile has four registration markers R at        known locations (i.e., points where x-corners are guaranteed to        occur due to the encoding scheme chosen), all have the same        orientations as each of the four tile corners), that facilitate        testing the hypothesis that the three features are at a tile        corner and 2 adjacent tile edges, respectively. A registration        marker is a portion of the tile that is constant no matter what        the encoded identity of the unique tile is. Thus, there are        pre-defined relationships between the elements of the 9×9        descriptor matrix. For example, in one implementation, the tile        corners (elements [0,0]; [8,0]; [0,8]; [8,8] and the        registration markers (elements [2,2]; [6,2]; [2,6]; [6,6]) are        all the x-corners with the same x-corner type (left-oriented or        right-oriented). Descriptors whose structure is inconsistent        with these pre-defined relationships are rejected.    -   g. Once the system verifies that the known relationships are        present, it can decode the encoded data in order to determine        the identity of the observed tile.

In the case where the tiles are not formed planar but, instead, aredefined or formed on a non-planar surface, e.g., the patterns are formedon a cylinder (FIG. 1) or a drill cone (FIG. 7), the above process ofassuming the entire tile's grid is deformed by an affine transformationis no longer applicable. Instead, the system assumes that the tile isonly locally-affine within a small sub-region of the tile, and variesfrom there in a smooth manner. The above steps (b)-(d) are modified toonly predict nearby grid locations, i.e., grid locations that are closeto a grid location where a grouping of x-corners has already beenpositively located. In one embodiment the nearby grid is within two gridunits (using L-infinity distance) of a located x-corner. At this point,the system assumes the pattern is smoothly-varying enough that theaffine assumption is valid when traversing the grid with only smallcorrections to the affine basis along the way. Thus, the system canprocess the planar tiles the same way as tiles that have curvature, bytraversing the grid and correcting the affine basis along the way. Onplanar tiles, the corrections will effectively be zero. The basisvectors are computed about each subset of detected defined points inorder to correct deviations from a purely affine assumption.

Once a set of descriptors has been computed for an image being analyzed,each descriptor is compared to a library of descriptors that are storedin the system and associated with a specific tile. For example, asdiscussed above, the matrix may include for each element −1 forleft-oriented x-corner, 0 for no x-corner, 1 for right-orientedx-corner. In one embodiment, since each descriptor can be associatedwith several potential unique tiles, a score is calculated between eachdetected descriptor and each library descriptor, and the highest-scoringlibrary matches are stored for each detected descriptor. The top scorescan be processed further to determine the tile by using additionalrelevant information, e.g., where certain points should be located.

In an embodiment, the system includes or has access to a database ofmodels of tracking patterns formed from one or more tiles. The presentinvention contemplates that the models can fall into two distinctarrangements of models. In the first arrangement, all the stored modelshave a unique subset of tiles where no tiles are repeated betweenmodels. In this case, knowing a single tile determines which modelyou're using. Each model is unique such that there is no replication ofthe arrangement of tiles between models. As such, the identification ofthe tile postulates a model pose. That is, each model in the modellibrary contains a set of tiles that are members of the model.

In a second arrangement of models, a number of models would share thesame tiles, but in different arrangements. As such, the system mustgenerate a hypothesis for each model of which the detected tile is amember. In this case, detecting two or more tiles would help prove thecorrectness of the model. In either arrangement, since noise and otherfactors might impact the detection of x-corners, the particular modelmust be further analyzed (tested) as discussed below to confirm themodel.

For each tile in a model, the database includes the 3D model locationsfor each point on the grid where defined points should appear. Theidentification of the tile or tiles in the image permits the system toselect the model that applies to the image being observed, and allows acorrespondence to be determined between at least the four tile cornersin the image coordinates and the four 3D model locations of the tilecorners. Through a conventional process of 3D pose estimation, thesystem estimates a rigid-body transform that defines the spatialrelationship of the model in a camera-centric coordinate system fromthese at least four correspondences.

The system then preferably applies the remaining tiles in the selectedmodel onto the image using the estimated rigid-body transform. Theseadditional tiles are tested against the tile identification hypotheses,and a count of the number of hypotheses consistent with a givencombination of model and rigid-body transform is aggregated. Only amodel with a number of positively-identified tiles that exceed somethreshold, for example, three correctly identified tiles, would beconsidered the proper model.

Once each camera reaches the end of this processing step, it is knownwhich image defined points (and, consequently, which 2D image locations)correspond to which model defined points (and, consequently, which 3Dmodel locations). Once both cameras have determined thesecorrespondences, determining stereo feature correspondences is a matterof matching image features that correspond to common model definedpoints. This can be accomplished using known techniques. There is noneed to apply epipolar constraints or pruning the resulting set ofcorrespondences. This is because the defined points are positivelyidentified with limited potential for spurious identification of amodel, and no false correspondences.

As described above, using the transform the system is able to uniquelyidentify the model based on the defined points 402 on the trackingpatterns 210, 308. Once that is performed, the stereo reconstruction isperformed by triangulating the corresponding pair of image definedpoints using known techniques. This is shown in steps 1100, 1110, 1120in FIG. 13. However, only image correspondences that are known to begood are passed in as input, and the association between reconstructed3D points (in stereo tracker coordinates) and 3D model points is passedthrough this step. The output of lookup matching (step 1110) provides a1:1 association between a set of pixel locations in the left image and aset of pixels in the right image. Through standard triangulationtechniques using the known arrangement of the two cameras, eachleft/right pair of pixel locations are triangulated to generate anestimate of the 3D location of that feature in the scene. Each of these3D coordinates are determined in a coordinate system fixed to the stereotracking system (e.g., the left camera or right camera location, or thecenter between the cameras, can be used to define the origin and axes ofthe coordinate system). In contrast the 3D model points are defined in amodel-centric coordinate system (e.g., the cone axis is the z axis, thecenter of the small end is (0,0,0).) Absolute orientation determines thetransform between these two coordinate systems (tracker-centric andmodel-centric).

Once at least three correspondences between specific 3D tracker points(i.e., points in the tracker-centric coordinate system) and specific 3Dmodel points (i.e., points in the model-centric coordinate system) areknown (step 1130), conventional absolute orientation processes (step1140) are used to determine the rigid-body transformation relating thetracker coordinate system to the model coordinate system, therebydetermining the spatial location and orientation of the model in trackercoordinates (step 1150). As such, the pose of the tile 400 and thetracking patterns 210, 308 are tied to the model. The data is then usedby the system to depict the actual movement of the oral fixture and toolfixture as movement of the associated models relative to scannedrepresentation of the area of interest (e.g. the prior scanned image ofthe oral cavity).

The processes for forming the oral fixture 12, for scanning the locationof fiducials on the fixture 12, and for registering the prior scannedimage to actual video image are described in detail in U.S. patentapplication Ser. No. 14/209,500. Once the oral fixture 12 is formed, thebracket assembly 100 is attached to the flanges 28, 30 on the oralfixture 12 and to the fixture tracking assembly 200. The oral fixture 12is attached to the appropriate teeth of the patient.

Referring to FIG. 14, in order to determine the location of the oralfixture 12 on the patient and the surgical tool 302, the presentinvention uses two external cameras 800 mounted in a location to viewthe fixture tracking pattern 210 and tool tracking pattern 308 anddetect the defined points as described above. The data from the cameras800 is transmitted to a processor 810 which conducts some or all of theprocessing described above and illustrated into FIG. 13. From that thesystem determines the position (pose) of the tracking patterns and theirmovement within a predetermined coordinate system. The system uses thelocation of the fiducial markers on the oral fixture 12 from the scannedimage and their relationship to the fixture tracking assembly 200 fordetermining movement of the patient and the location of the tool fixtureassembly 300 relative to the oral fixture 12, and then to calculate thelocation of the tool bit tip relative to the operation site.

The present invention provides significant advantages over the priorexisting stereo tracking systems. First, the present inventionpreferably implements a significant number of computationally-expensivesteps on each camera independently of the other cameras and the mainprocessing system. This allows for easier scaling of the system,especially as the number of cameras in the system grows beyond two. In aconventional stereo tracking system the requirement of featurecorrespondence would grow as a function of O(Nc²) where Nc is the numberof cameras used in a standard stereo tracking system.

It is contemplated that the processing could be carried out in aprocessor in the camera and the programming and data could be embeddedin memory associated with the processor.

These cameras could be placed remotely on a distributed network. Theresulting communication bandwidth would be a tiny fraction of thepassing the images themselves, or even the set of image feature pointsthat are required in conventional systems.

The rich nature of the identified tiles makes the potential for spuriousidentification of a model exceedingly remote, whereas significantnumbers of features detected on non-model objects in the standard stereotracking case can give rise to many spurious model identifications.

While the above description refers to the term “tile” as auniquely-identifiable unit, which can be arranged to form an opticalpattern, the term is not restricted to the conventional notion of“tiling” of such units as abutting and non-overlapping. Co-pendingapplication Ser. No. 14/209,500 details an interleaved encoding schemewhere multiple tiles overlap and occupy the same portion of a pattern inorder to enhance two-scale detection. It is contemplated that even in aconventionally-tiled pattern, the arrangement of the unique tiles can bechosen such that each junction of 4 tiles forms another unique tile fromthe combination of portions of the tiles that are nearest the junction,in such a way that every patch on the pattern is a member of 2 or moretiles. Such a tiling would have the advantage that when portions of thepattern are obscured from view, a greater number of complete tilesshould be visible to aid model identification. While the abovedescription details tile boundaries with 90-degree corners it is furthercontemplated that the tile boundaries can contain arbitrary polyline orrounded segments. The two-scale encoding scheme in co-pendingapplication Ser. No. 14/209,500 includes a combination of square tilesand complex tiles that have holes.

The calculations and programming techniques used for tracking anddetermining the motions of the various components are well known and,thus, no further information is necessary.

The foregoing embodiments are based on the assumption that the patienthas sufficient teeth to mount the oral fixture 12 and fixture trackingassembly 200. If, however, the condition of the patient's mouth preventsattachment of either or both of the oral fixture 12 and fixture trackingassembly 200, the present invention envisions that either component canbe directly mounted to the jaw bone of the patient.

While the above description refers to a surgical tool or instrument thatincludes a drill, the term “surgical instrument” or “surgical tool” isintended to cover other tools used during intraoral procedures, such asablation tools for ablating tissue, including third molars in children.

The system or systems described herein may be implemented on any form ofcomputer or computers and the components may be implemented as dedicatedapplications or in client-server architectures, including a web-basedarchitecture, and can include functional programs, codes, and codesegments. The system of the present invention may include a softwareprogram be stored on a computer and/or storage device (e.g., mediums),and/or may be executed through a network. The method may be implementedthrough program code or program modules stored on a storage medium.

For the purposes of promoting an understanding of the principles of theinvention, reference has been made to the preferred embodimentsillustrated in the drawings, and specific language has been used todescribe these embodiments. However, no limitation of the scope of theinvention is intended by this specific language, and the inventionshould be construed to encompass all embodiments that would normallyoccur to one of ordinary skill in the art.

The embodiments herein may be described in terms of various processingsteps. Such processing steps may be realized by any number of hardwareand/or software components that perform the specified functions. Forexample, the described embodiments may employ various integrated circuitcomponents, e.g., memory elements, processing elements, logic elements,look-up tables, and the like, which may carry out a variety of functionsunder the control of one or more microprocessors or other controldevices. Similarly, where the elements of the described embodiments areimplemented using software programming or software elements theinvention may be implemented with any programming or scripting languagesuch as C, C++, Java, assembler, or the like, with the variousalgorithms being implemented with any combination of data structures,objects, processes, routines or other programming elements. Functionalaspects may be implemented in algorithms that execute on one or moreprocessors. Furthermore, the embodiments of the invention could employany number of conventional techniques for electronics configuration,signal processing and/or control, data processing and the like. Thewords “mechanism” and “element” are used broadly and are not limited tomechanical or physical embodiments, but can include software routines inconjunction with processors, etc.

The particular implementations shown and described herein areillustrative examples of the invention and are not intended to otherwiselimit the scope of the invention in any way. For the sake of brevity,conventional electronics, control systems, software development andother functional aspects of the systems (and components of theindividual operating components of the systems) may not be described indetail.

Finally, the steps of all methods described herein are performable inany suitable order unless otherwise indicated herein or otherwiseclearly contradicted by context. The use of any and all examples, orexemplary language (e.g., “such as”) provided herein, is intended merelyto better illuminate the invention and does not pose a limitation on thescope of the invention unless otherwise claimed. Numerous modificationsand adaptations will be readily apparent to those skilled in this artwithout departing from the spirit and scope of the invention.

We claim:
 1. An image guidance system for tracking a surgical instrumentduring oral surgery comprising: a fixture configured to be removablyattached to a patient's anatomy in a location spaced apart from asurgical area; a first tracking assembly including a bracket assemblyremovably attached to the fixture and a first tracking pattern surface,the first tracking pattern surface including a first optically visiblepattern, the bracket assembly positioning the first tracking pattern ata location spaced apart from the surgical area; a tool for use in thesurgical procedure, the tool having a proximate tip end to which a drillbit is attached, an intermediate portion adapted to be held by a user,and a distal end opposite from the tip end; a second tracking assemblyincluding a second tracking pattern surface mounted to the tool at alocation spaced apart from the tip end of the tool so as to be spacedapart from the surgical area and positioned external to the oral cavityduring use, the second tracking pattern surface including a secondoptically visible pattern; a plurality of cameras mounted at a locationspaced apart from the surgical area and the first and second trackingassemblies at a position that permits the cameras, when activated, tocapture images of the optically visible patterns on the first and secondtracking pattern surfaces; a reference dataset including the locationand orientation of the fixture with respect to a CT scan of the surgicalarea; a processing system to process the captured images and configuredto recognize the optically visible patterns and triangulate thelocations and orientations of the first and second tracking assemblies,the processing system determining the location and orientation of thetracked tool based on the reference dataset and analyzing relativetransforms between each camera of the optically visible patterns on thefirst and second tracking pattern surfaces.
 2. An image guidance systemaccording to claim 1 wherein the fixture is configured to removablyattach to opposite sides of one or more teeth in a patient's mouthwithout the need for tools for attaching and removing the fixture.
 3. Animage guidance system according to claim 1 wherein the bracket assemblyincludes a bracket mount that removably attaches to a flange on thefixture, a tracking mount that attaches to the first tracking patternsurface, and a support arm that attaches the bracket mount to thetracking mount.
 4. An image guidance system according to claim 3 whereinthe bracket mount includes two spaced apart mounting posts that engagewith the flange.
 5. An image guidance system according to claim 3wherein the attachment of the tracking mount to the first trackingpattern surface is adjustable, the adjustable attachment of the trackingmount permitting angular adjustment of the first tracking patternsurface relative to the support arm.
 6. An image guidance systemaccording to claim 3 wherein the tracking mount includes a base with aseries of indentations and protrusions.
 7. An image guidance systemaccording to claim 6 wherein the first tracking assembly includes aframe with an attachment to the tracking mount, the attachment beingconfigured to permit the frame to be adjustable with respect to thebracket assembly.
 8. An image guidance system according to claim 7wherein the frame includes a series of indentations and protrusions thatare configured to mate with the indentations and protrusions on thetracking mount so as to permit the rotational orientation of thetracking frame relative to the base to be adjustable.
 9. An imageguidance system according to claim 1 wherein the optically visiblepatterns each contain a plurality of 2D contrasting shapes, thecontrasting shapes arranged so as to uniquely differentiate eachoptically visible pattern from the other optically visible pattern. 10.An image guidance system according to claim 9 wherein each camera islocated so as to capture at least a portion of the optically visiblepattern on each of the first and second tracking pattern surfaces suchthat the image of each optically visible pattern captured by the camerais at a different viewing angle than the image of the same opticallyvisible pattern captured by the other camera(s), the cameras configuredto send to the processor data representative of the 2D images of theoptically visible patterns captured by the cameras.
 11. An imageguidance system according to claim 10 wherein the processor includes adataset of patterns, including location data for contrasting shapes oneach of the patterns in the dataset, and wherein the processor isconfigured to analyze the dataset of patterns to determine which of thepatterns in the dataset corresponds to each of the optically visiblepatterns in the 2D images received from the cameras.
 12. An imageguidance system according to claim 11, wherein the contrasting shapesinclude adjacent boxes of distinct color, each set of adjacent boxesbeing separated from one another by a line, and wherein a defined pointis formed by a center point of two intersecting lines, and wherein theprocessor determines the location of defined points in the dataset ofpatterns and the optically visible patterns for determining whether agiven pattern from the dataset of patterns corresponds to the opticallyvisible patterns.
 13. An image guidance system for tracking a surgicalinstrument during oral surgery comprising: a fixture configured to beremovably attached to a patient's anatomy in an oral cavity at alocation spaced apart from a surgical area; a first tracking assemblyincluding a bracket assembly having a first end adjustably attached tothe fixture and extending away from the oral cavity, the bracketassembly having a second end attached to a first tracking patternsurface, the bracket assembly permitting the first tracking pattern tobe adjustable with respect to the fixture, the first tracking patternsurface including a first optically visible pattern, the bracketassembly positioning the first tracking pattern at a location spacedapart from the oral cavity; a surgical tool for use in the surgicalprocedure, the tool having a proximate tip end, an intermediate portionadapted to be held by a user, and a distal end opposite from the tipend; a second tracking assembly including a second tracking patternsurface, the second tracking assembly configured to mount to a surgicaltool for use in the oral surgery at a location spaced apart from the tipend of the tool so as to be spaced apart from the surgical area andpositioned external to the oral cavity during use, the second trackingpattern surface including a second optically visible pattern; aplurality of cameras mounted at a location spaced apart from thesurgical area and the first and second tracking assemblies at a positionthat permits the cameras, when activated, to detect images of theoptically visible patterns on the first and second tracking patternsurfaces; a reference dataset including the location and orientation ofthe fixture with respect to a CT scan of the oral cavity; and aprocessing system to process the captured images and configured torecognize the optically visible patterns and triangulate the locationsand orientations of the first and second tracking assemblies, theprocessing system determining the location and orientation of thetracked tool based on the reference dataset and analyzing relativetransforms between each camera of the optically visible patterns on thefirst and second tracking pattern surfaces.
 14. An image guidance systemaccording to claim 13 wherein the fixture is configured to removablyattach to opposite sides of one or more teeth in a patient's mouthwithout the need for tools for attaching and removing the fixture,wherein the bracket assembly includes a bracket mount that removablyattaches to a flange on the fixture, a tracking mount that attaches tothe first tracking pattern surface, and a support arm that attaches thebracket mount to the tracking mount, wherein the attachment of thetracking mount to the first tracking pattern surface is adjustable, theadjustable attachment of the tracking mount permitting angularadjustment of the first tracking pattern surface relative to the supportarm.
 15. An image guidance system according to claim 14 wherein thetracking mount includes a base with a series of indentations andprotrusions, the first tracking assembly including a frame with anattachment to the tracking mount, the attachment being configured topermit the frame to be adjustable with respect to the bracket assembly,and wherein the frame includes a series of indentations and protrusionsthat are configured to mate with the indentations and protrusions on thetracking mount so as to permit the rotational orientation of thetracking frame relative to the base to be adjustable.
 16. An imageguidance system according to claim 13 wherein the optically visiblepatterns each contain a plurality of 2D contrasting shapes, thecontrasting shapes arranged so as to uniquely differentiate eachoptically visible pattern from the other optically visible pattern, andwherein each camera is located so as to capture at least a portion ofthe optically visible pattern on each of the first and second trackingpattern surfaces such that the image of each optically visible patterncaptured by the camera is at a different viewing angle than the image ofthe same optically visible pattern captured by the other camera(s), thecameras configured to send to the processor data representative of the2D images of the optically visible patterns captured by the cameras. 17.An image guidance system according to claim 16, wherein the processorincludes a dataset of patterns, including location data for contrastingshapes on each of the patterns, and wherein the processor is configuredto analyze the dataset of patterns to determine which of the patterns inthe dataset corresponds to each of the optically visible patterns in the2D images received from the cameras.
 18. An image guidance systemaccording to claim 17, wherein the contrasting shapes include adjacentboxes of distinct color, each set of adjacent boxes being separated fromone another by a line, and wherein a defined point is formed by a centerpoint of two intersecting lines, and wherein the processor determinesthe location of defined points in the dataset of patterns and theoptically visible patterns for determining whether a given pattern fromthe dataset of patterns corresponds to the optically visible patterns.19. An image guidance system according to claim 1, wherein the secondtracking pattern surface is mounted to an external portion of the toolso as to position the visible pattern in a field of view of the cameras.